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PRTVT TMTN ARY AMENDMENT 

Assistant Commissioner for Patents 
Washington, D.C. 20231 

Sir: 

Prior to an examination on the merits, please amend the above-referenced application as 
follows: 

IN THE SPECIFICATION. 

Page 25, line 28, change "ateradian filed-of-view" to -steradian field-of-view-. 
TN THE CLAIMS: 

Please cancel claims 1-35 without prejudice to, or disclaimer of, the subject matter recited 

therein. 

Please add new claims 36-62 as follows: 

36. (New) A method comprising the step of transforming a selected portion of a spherical 
image to generate a perspective-corrected planar image portion using a computer in real time in 
response to user input to the computer. 
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37. (New) The method of claim 36, wherein the user input includes information 
comprising an identification of the selected portion of the spherical image. 

38. (New) The method of claim 36, further including the step of displaying at least a 
portion of the perspective-corrected planar image portion on a display. 

39. (New) The method of claim 36, further including the step of performing at least one 
of rotating, tilting, and zooming the perspective-corrected planar image portion in real time in 
response to the user input. 

40. (New) The method of claim 36, further including the step of changing a location of 
the selected portion within the spherical image in real time in response to the user input. 

41. (New) The method of claim 36, wherein the selected portion includes a plurality of 

non-adjacent sub-portions. 

42. (New) The method of claim 36, wherein the step of transforming is performed 
according to at least one calibration coefficient, the at least one calibration coefficient being 

controlled by the user input. 

43 . (New) The method of claim 36, wherein the spherical image is a moving picture video 
image comprising a plurality of still-image spherical image frames, the method further including the 
step of outputting the perspective-corrected planar image at a real-time video rate compatible with 

commercial video standards. 

44. (New) The method of claim 36, further including the step of rotating the perspective- 
corrected planar image portion such that the perspective-corrected planar image portion is vertically 

aligned with a gravity vector. 

45. (New) A method for creating a spherical image comprising the steps of: 
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capturing a first approximately hemispherical image via a first camera including a first 
approximately hemispherical lens directed in a first direction; 

waiting for a period of time after capturing the first approximately hemispherical image; 

capturing a second approximately hemispherical image after the period of time is complete 
via a second camera including a second approximately hemispherical lens directed in a second 
direction approximately opposite to the first direction; and 

combining the first approximately hemispherical image with the second approximately 
hemispherical image to form the spherical image. 

46. (New) The method of claim 45, wherein the second camera is the first camera. 

47, (New) The method of claim 45, wherein the first approximately hemispherical image 
and the second approximately hemispherical image are captured on a same frame of film. 

49. (New) The method according to claim 45, wherein the first approximately 
hemispherical image is a greater than 180° field-of-view image and the second image is a greater than 
180° field-of-view image. 

50. (New) The method according to claim 45, wherein the first camera includes film 
configured to be exposed with the first approximately hemispherical image, the method further 
comprising the step of converting the first approximately hemispherical image exposed on the film 
into an electronic format. 

5L (New) Apparatus for creating a spherical image, the apparatus comprising: 

a first approximately hemispherical lens directed in a first direction for receiving a first 

approximately hemispherical image; 

a second approximately hemispherical lens directed in a second direction approximately 

opposite to the first direction for receiving a second approximately hemispherical image, the 



McCall et al. - Atty. Dkt No. 01096.81 1 10 

apparatus being configured to capture the second approximately hemispherical image a period of time 
after capturing the first approximately hemispherical image; and 

a computer for combining the first approximately hemispherical image with the second 
approximately hemispherical image to form the spherical image. 

52. (New) The apparatus of claim 5 1, further including a camera coupled to the computer 
for capturing the first and second approximately hemispherical images, the first approximately 
hemispherical lens being optically connected to the camera via first reflective optics and the second 
approximately hemispherical lens being optically connected to the camera via second reflective optics. 

53. (New) The apparatus according to claim 52, wherein the camera comprises a still 
camera. 

54. (New) The apparatus according to claim 52, wherein the camera comprises a video 
camera. 

5 5 . (New) The apparatus according to claim 52, wherein the camera comprises a motion 
picture camera. 

56. (New) The apparatus according to claim 52, wherein the camera comprises a linear 
scanning CID camera array. 

57. (New) The apparatus according to claim 52, wherein the camera comprises a linear 

scanning CCD camera array. 

58. (New) The apparatus according to claim 52, wherein the camera comprises a linear 

scanning CMOS APS camera array. 

59. (New) The apparatus according to claim 51, further including a first camera for 
capturing the first approximately hemispherical and a second camera for capturing the second 
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approximately hemispherical image, the first and second cameras being attached to each other in a 
back-to-back configuration. 

60. (New) The apparatus according to claim 52, wherein the first approximately 
hemispherical lens is coupled to the camera via a fiber optic line, the first approximately hemispherical 
image received by the first approximately hemispherical lens being transferred to the camera via the 
fiber optic line. 

61 . (New) The apparatus according to claim 51, further including at least one of a film 
for capturing the first and second approximately hemispherical images on a same frame of the film. 

62. (New) The apparatus according to claim 51, wherein the first approximately 
hemispherical image is a greater than 180° field-of-view image and the second image is a greater than 
180° field-of-view image. -- 

REMARKS 

Prior to an examination on the merits, it is respectfully requested that the present Amendment 
be entered in the above-identified application. Claims 36-62 are now pending. 



Respectfully submitted, 



Dated: May 21, 1999 
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This invention relates generally to an apparatus and method 
for capturing an image having a spherical f ield-of -view for 
subsequent viewing. Specifically, the present invention relates 
to a system involving a single camera having a lens with a field 
of view of at least 180° and associated method for capturing a 
first hemispherical image for subsequent combination into the 
spherical image. Alternatively, when the system comprises two 
cameras with such lenses mounted in a securely attached back-to- 
back arrangement, the system and method captures two distinct 
hemispherical images for subsequent combination into the 
spherical image. The preferred system includes a single-use, 
still image camera. 
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Background Art 

The discussion of the background art related to the 
invention described herein relates to two subjects: spherical 
image capture and subsequent captured image transformations. 

Spherical Image Capture 

The goal of imaging technology is to make the observer feel 
as though he or she is part of the image. Prior art systems have 
partially accomplished this goal. Unfortunately, the ability of 
prior art systems to make the user feel part of the captured 
images are proportional to the cost of the image capture system. 

Relating to inexpensive image capturing systems, camera 
companies have introduced disposable cameras, A disposable 
camera generally refers to a single-use camera that includes 
film, a lens, and a camera body, all in a single compact shell. 
The film includes either a single frame of film or multiple 
frames of film. After the entire roll of film has been exposed, 
the entire camera is returned for film developing. All the 
photographer receives back are the developed prints or slides. 
The manufacturer then recycles the parts from the returned 
camera, adds film, and ships the camera to a retailer for sale 
again. Disposable cameras come in various types including 
regular magnification cameras, telephoto cameras, water resistant 
cameras, and panoramic cameras. 

Images captured by panoramic cameras provide wide angle 
horizontal images (left to right) but lack wide angle vertical 
images (up and down) . Accordingly, while capturing a wide f ield- 
of -view on one plane (horizontal) , the photographer loses the 
wide f ield-of -view on the other plane (vertical) . Rotating the 
camera only alters the wide angle direction. The following 
example illustrates this shortcoming. Suppose a photographer 
desires to capture the grandeur of a dense forest from within the 
forest. While an image captured by a panoramic camera would 
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include a sweeping cross section of trees (left to right) , it 
would only include, at most, the middle portions of the nearest 
trees . To capture the forest floor and canopy, the photographer 
would have to take multiple panoramic photographs from looking 
5 almost straight down to looking straight up. The final image of 
the forest would then only be realized with the laborious task of 
manually cutting and pasting the different images together. 
Unfortunately, the left and right ends of the final image become 
distorted and cannot be easily resolved. The distortions created 
10 are similar to those encountered in map-making where one tries to 
represent a round earth on a flat map. Specifically, objects and 
relative distances near the extremes of the wide angle image 

r'i 

It become distorted. Additionally, this approach wastes film. 

A slightly more complex panoramic camera employs a scanning 
rljl5 drive mechanism which selectively exposes vertical strips of film 
45 as the camera scans from extreme to extreme. However, scanning 

^ panoramic cameras invariably introduce noise into captured images 

« through vibrations generated from their scanning motions as well 

£J as take a relatively long period of time to capture the image. 

: "-;20 Other wide-angle image capturing systems exist. For 

H example, IMAX and 70 mm films provide high definition images on a 

^ large screen. However, these screens are flat. While a viewer 

can feel part of the scene when staring straight ahead, this 
feeling dissipates where the screen ends. 
25 Another imaging system includes the OMNIMAX camera and 

projection system where an image was recorded and later projected 
on a spherical screen to produce an image 180 degrees wide, 100 
degrees up from the horizon and 20 degrees below. While this 
system offers significant improvements over a flat screen 
30 projection system, the viewer's absorption into the displayed 
images is limited by the edges of the displayed image. 

Another image capture and display system is U.S. Patent No. 
5,023,725 to McCutchen. McCutchen discloses a dodecahedral 
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imaging system which breaks a sphere into 12 discrete 
polyhedrons. Each section has its own dedicated CCD camera. The 
images are captured and displayed on the walls of a hemispherical 
room. This system offers increased resolution through increasing 
5 the number of cameras used. However, as the number of cameras 
increase, the bulk of the imaging system likewise increases. 
Additionally, each camera has to be perfectly aligned with 
respect to the other cameras to adequately capture a spherical 
image. Using McCutcheon's system, increased resolution requires 
10 more bulk and more expense. Furthermore, the images of each 
camera are not integrated together. Accordingly, the system 

™ . fails to account for the seams between the displayed images. 

<B While quickly moving images may mask these edge effects, the edge 

r effects may be more noticeable with slow moving images. 

LT-5 

'i£ Captured Image Transformations 

■Sj Camera viewing systems are used in abundance for 

surveillance, inspection, security, and remote sensing. Remote 
ti viewing is critical, for example, for robotic manipulation tasks, 

fkbo Close viewing is necessary for detailed manipulation tasks while 
r 1 wide-angle viewing aids positioning of the robotic system to 

3 avoid collisions with the work space. Most of these systems use 

either a fixed-mount camera with a limited viewing field to 
reduce distortion, or they utilize mechanical pan-and-tilt 
25 platforms and mechanized zoom lenses to orient the camera and 

magnify its image. In the application where orientation of the 
camera and magnification of its image are required, the 
mechanical solution is large in size and can subtend a 
significant volume making the viewing system difficult to conceal 
3 0 or use in close quarters. Several cameras are usually necessary 
to provide wide-angle viewing of the work space. 

In order to provide a maximum amount of viewing coverage or 
subtended angle, mechanical pan/tilt mechanisms usually use 
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motorized drives and gear mechanisms to manipulate the vertical 
and horizontal orientation. An example of such a device is shown 
in U.S. Patent Number 4,728,839 issued to J.B. Coughlan, et al, 
on March 1, 1988. Collisions with the working environment caused 
5 by these mechanical pan/tilt orientation mechanisms can damage 

both the camera and the work space and impede the remote handling 
operation. Simultaneously, viewing in said remote environments 
is extremely important to the performance of inspection and 
manipulation activities. 

10 Camera viewing systems that use internal optics to provide 

wide viewing angles have also been developed in order to minimize 
the size and volume of the camera and the intrusion into the 
viewing area. These systems rely on the movement of either a 
mirror or prism to change the tilt-angle of orientation and 

15 provide mechanical rotation of the entire camera to change the 

pan angle of orientation. Additional lenses are used to minimize 
distortion. Using this means, the size of the camera orientation 
system can be minimized, but "blind spots" in the center of the 
view result. Also, these systems typically have no means of 

20 magnifying the image and or producing multiple images from a 
single camera. 

Further, references that may be relevant to the evaluation 
of the captured image transformations as described herein include 
U.S. Patent Numbers 4,772,942 issued to M. J. Tuck on September 
25 20, 1988; 5,067,019 issued to R. D. Juday on November 19, 1991; 
and 5,068,735 issued to K. Tuchiya, et al on November 26, 1991. 

Objects of the Invention 

Accordingly, it is an object of the present invention to 
3 0 provide an apparatus that captures at least one hemispherical 
image for later manipulation. 
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Another object of the invention is to provide an apparatus 
which captures a spherical image from two images produced by two 
cameras . 

Another object of the invention is to form a single 
spherical image from the captured image or images. 

It is a further object of the invention to provide a 
spherical image capture system and method without the bulk of a 
large number of cameras and the necessity of multiple camera 
alignment . 

Another object of the invention is to reduce the number of , 
seams in a formed image* 

Another object of the invention is to accomplish the above 
objectives using a single-use, disposable camera. 

Another object of the invention is to provide a system for 
displaying a complete spherical image with perspective correction 
and without edge effects and image distortion. 

Another object of the invention is to enable interaction 
with any portion of the spherical image with the selected portion 
being perspective corrected. 

It is another object of the present invention to provide 
horizontal orientation (pan) , vertical orientation (tilt) and 
rotational orientation (rotation) of the viewing direction with 
no moving mechanisms. 

It is another object of the present invention to provide the 
ability to magnify or scale the image (zoom in and out) 
electronically. 

It is another object of the present invention to provide 
electronic control of the image intensity (iris level) . 

It is another object of the present invention to be able to 
accomplish pan, tilt, zoom, rotation, and iris adjustment with 
simple inputs made by a lay person from a joystick, keyboard 
controller, or computer controlled means. 



Application of McCall et ai. 

It is also an object of the present invention to provide 
accurate control of the absolute viewing direction and 
orientations using said input devices. 

A further object of the present invention is to provide the 
ability to produce multiple images with different orientations 
and magnifications simultaneously from a single input image. 

Another object of the present invention is to be able to 
provide these images at real-time video rate, e.g. thirty 
transformed images per second, and to support various display 
format standards such as the National Television Standards 
Committee RS-170 signal format and/or higher resolution formats 
currently under development and to provide the images to a 
computer display performing perspective correction transforms on 
a personal computer system. 

It is also an object of the present invention to provide a 
system than can be used for automatic or manual surveillance of 
selected environments, with optical views of these environments 
corrected electronically to remove distortion so as to facilitate 
this surveillance. 

It is another object of this invention to provide a means 
for directly addressing each picture element of an analog image 
captured with an imaging device having a f ield-of -view, the 
picture elements being addressed in a non-linear sequence 
determined in a manner similar to that described by U.S. patent 
no. 5,185,667 to provide a distortion-corrected image without 
requiring the use of filters and memory holding buffers. 

Another object of the present invention is to provide a 
means for directly addressing each picture element of an image 
(still or video) captured using an imaging device having a two- 
dimensional f ield-of -view. 
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Summary of the Invention 

According to the principles of the present invention, at 
least one camera with a 180° or greater f ield-of -view lens 
captures a spherical image. When the system employs two cameras 
with such lenses, the cameras and lenses are mounted in a back- 
to-back arrangement. When used in this disclosure and attached 
claims, "back-to-back" means two cameras clasped together such 
that the image planes of the lenses fall between each of the 
lenses and both lenses' optical axes are collinear with a single 
line which passes through each lens and camera. An imaging 
element or elements capture the images produced by the lenses. 
When used herein and in the claims, an "imaging element" or 
"imaging elements" refer to both film and linear scanning devices 
and alternatives thereof upon which an image is focused and 
captured. The captured images from each camera are stored and 
combined to form a single, spherical image (a final, formed 
image) . When used herein and in the claims, "stored" not only 
means to digitally store an image in a retrievable form but also 
means to capture the image on film. To form the spherical image, 
the system includes a converter which identifies, joins, and 
smooths the edges (also referred to as the "seams") of each 
hemispherical image. When used herein and in the claims, a 
"converter" refers to not only a manual system (splicing by hand 
and airbrush image altering techniques) but also an automatic 
image processing system (digital processing by a computer where 
images are altered automatically) for combining the two images . 
together. Where a partial overlap exists between the two 
hemispherical images, the converter processes the partial overlap 
to remove the overlap and any distortion and create a single, 
complete, formed spherical image. Finally, a selected planar 
portion of the spherical image may be displayed on a personal 
computer using perspective correction software or hardware. 
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A method for capturing a spherical image includes the steps 
of capturing a first hemispherical image with a first camera 
including a first 180° or greater f ield-of -view lens; receiving a 
second hemispherical image either by capturing the second 
hemispherical image by means of a second camera including a 
second oppositely directed 180° or greater f ield-of -view lens or 
by creating a mirror image of the first hemispherical image; and, 
combining the first and second oppositely directed hemispherical 
images to create a spherical image. 

An apparatus capturing a spherical image includes a first 
camera equipped with a 180° or greater f ield-of -view lens, the 
first camera and the lens directed in a first direction, the 
first camera capturing a first image; a second device either 
forming a second image corresponding to a mirror image of the 
first image or including a second camera equipped with a 180° or 
greater f ield-of -view lens, directed in a second direction 
opposite to the first direction, the second camera capturing the 
second image; and, a combining system for combining the first and 
second images into a formed spherical image . 

The cameras disclosed above capture high resolution images. 
Various cameras may be used including still cameras, video 
cameras, and CCD, CID, or CMOS APS cameras. With high resolution 
(crystal clear) images as a goal, the system employs a still 
camera capturing a high resolution image on a fine grain film. 
Film generally composes a layer of silver halide crystals. Upon 
exposure to light, this silver halide layer picks up the image 
exposed to it. The greater the number of separate halide 
crystals, the greater the resolution of the film. Thus, a finer 
grain size refers to an increase in number of silver halide 
crystals per unit area of film which in turn refers to an 
increase in the potential resolution of the film medium. 

When capturing a spherical image with two single-use 
cameras, the cameras include additional features allowing for 
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dual image capture. Where "single-use camera" is referred to 
herein and in the claims, it refers to a disposable camera or 
other alternative. The additional features which aid in 
spherical image capture include attachment devices which attach 
the backs of the cameras to each other. When used herein, 
"attachment devices" refer to locking pins, locking clasps, lever 
and hook systems, and alternatives thereof. Also, each camera's 
shutter release may be controlled by a single button (common 
shutter release control) with either a mechanical or electrical 
servo linkage releasing each camera's shutter. Additionally, to 
allow a photographer to avoid his or her image from being 
captured by the spherical image capture system, the dual camera 
system includes a shutter auto timer or a remote shutter 
activation control controlling the common shutter release 
control. The remote shutter control may be an IR transmitter or 
remote shutter release cable. Further, the dual camera system 
may include two different shutters operable independently or 
sequentially. The sequential shutter operations allow the 
photographer to walk around to the other side of the dual camera 
system so as not to become part of the captured spherical image. 

According to the present invention, when using a still image 
recorded on film, after developing the film, a high resolution 
digital scanner scans and digitizes the image contained in the 
developed film and stores the digitized image in a retrievable 
medium. The retrievable medium includes, inter alia, CD-ROMs, 
magnetic disks and tapes, semiconductor devices, and magneto- 
optical disks. 

As referred to above, the second image may be created from 
the first image. This may be accomplished by at least one of two 
methods: first, manually, by forming the second image by hand 
and, second, automatically, by means of a computer running image 
processing software. As to manually creating the image, the film 
developing and printing steps generate the second image. For 
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example, after printing or scanning the first hemispherical 
image, a technician or device flips or likewise reverses the film 
storing the at least hemispherical image (from left to right 
orientation to right to left orientation) and scans or prints the 
film again. 

The automatic printing or scanning technique creates the 
second hemispherical image (also known as a mirror image of the 
first image) through appropriate software. Alternatively, image 
processing software or hardware may reverse the scanned image 
without the need to manually flip a developed piece of film. 

The converter (automatic or manual) seams the two 
hemispherical images together and stores a generated, complete 
spherical image in a storage medium including CD-ROMs, magnetic 
disks and tapes, semiconductor devices and magneto-optical disks. 
This converting may be accomplished by sending the camera and/or 
film to a processing center which sends back the spherical image 
stored in one of the above storage mediums. 

Finally, using the perspective correction and manipulation 
system as disclosed in Patent No. 5,185,667 and its progeny 
including Patent Nos . 5,359,363 and 5,313,306 and Serial nos. 
08/189,585, 08/339,663, and 08/373,446, the formed, seamless, 
spherical image may be explored. These patents and applications 
and others herein are expressly incorporated by reference. 

Preferably, a personal computer system runs the perspective 
correction software or hardware. These computers may be directly 
linked to the image capturing system (allowing viewing of the 
spherical image as captured by the hemispherical camera or 
cameras and manipulated by the perspective correction system) or 
may remain completely separate (photographing an image, sending 
the film to a processing center which creates a spherical image 
from the photograph or photographs, and returning the spherical 
image stored in a retrievable form for display on a personal 
computer) . 
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Brief Description of the Drawings 

The above mentioned features of the invention will become 
more clearly understood from the following detailed description 
of the invention read together with the drawings in which: 

Figure 1 is a diagram of the fields of view for 180° and 
greater than 180° fields of view lenses as mounted to a single 
camera body. 

Figure 2 shows two back- to-back cameras each capturing more 
than 180° fields of view images. 

Figure 3 shows two back- to-back cameras each capturing 180° 
fields of view images. 

Figure 4 shows an alternated embodiment of the spherical 
capture system of the present invention. 

Figures 5A and 5B relate to the elements used to capture a 
spherical image. Figure 5A shows two hemispherical lenses 
capturing complementary hemispherical images and feeding them to 
remote cameras. Figure 5B shows a hemispherical lens capturing a 
hemispherical image and a mirror image converter for converting 
the first hemispherical image into a second hemispherical image. 

Figure 6A shows two hemispherical lenses similar to that of 
Figure 5A passing images to local cameras through reflective and 
refractive optics. Figure 6B shows two hemispherical lenses 
conveying images to a single camera. 

Figures 7A and 7B represent two hemispherical images 
combined into a single spherical image. 

Figure 8 shows a storage/display option of the instant 
invention. 

Figures 9A and 9B show a schematic block diagram of the 
signal processing portion of the present invention illustrating 
the major components thereof. Figure 9A shows the perspective 
correction process implemented in hardware. Figure 9B shows the 
perspective correction process implemented in software, operating 
inside a personal computer. 
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Figure 10 is an exemplary drawing of an at least 
hemispherical image used as input by the present invention. 
Lenses having other f ield-of -view values will produce images with 
similar distortion, particularly when the f ield-of -view is about 
5 eighty degrees or greater. 

Figure 11 is an exemplary drawing of the output image after 
correction for a desired image or orientation and magnification 
within the original image. 

Figure 12 is a schematic diagram of the fundamental geometry 
10 that the present invention embodies to accomplish the image 
transformation. 

^ Figure 13 is a schematic diagram demonstrating the 

w. projection of the object plane and position vector into image 

Y' J plane coordinates . 

yl5 

*k" Detailed Description of the Invention 

Spherical Image Capture 

f3 The disclosed Spherical Image Capture system employs the 

flfeo components disclosed in Figures 1-8 to capture hemispherical 
r * images and form spherical images. The image transform engine as 

Jl disclosed in Figures 9-13 operates to transform selected portions 

of the formed spherical images into planar, perspective corrected 

portions . 

25 Referring to Figure 1, camera 601 includes lens 602 with 

optical axis A, image plane I, and a f ield-of -view of 180° or 
greater. If lens 602 has a 180° f ield-of -view it captures at 
most the image from hemisphere 603. On the other hand, if lens 
602 has a f ield-of -view greater than 180°, then it captures the 

30 image from sector 604 (shown by dotted lines) as well as that of 
hemisphere 603 . 

Figure 2 shows a camera body 701 (which may include two 
cameras) connected to lenses 702 and 703 (with image planes I 702 
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and I 703 , respectively) . Each of lenses 702 and 703 have fields 
of view greater than 180°. Placed in a back-to-back arrangement 
where the lenses are mounted such that the image planes I 702 and 
I 703 from the lenses fall between each of the lenses and both 
5 lenses' optical axes A coincide in a single line which passes 
through each lens and camera, they capture the spherical image 
surrounding the camera body 701. It should be noted, however, 
that the thickness of the camera body 701 plays a role in how 
much of the spherical image surrounding the camera is captured. 
10 Specifically, the objects on the sides of the camera may or may 
not be completely photographed depending on their distances from 
the camera body 701. For example, if objects are within boundary 
704, some of the objects may fall into the camera's blind spots 
707 and not be completely photographed. On the other hand, 
73-5 because of the converging angles of lenses' greater than 180° 
fz fields of view, objects within sectors 705 will be photographed 

7r\ twice: first, by means of the image captured by lens 702 and, 

& second, by means of the image captured by lens 703 . Decreasing 

ri the distances between the lenses reduces blind spots 707 of the 

Rj2 0 spherical capture system. In this example, reducing the distance 

between the lenses means reducing the thickness of the camera 
J} body 701. Reducing the camera body thickness can be 

accomplished, for example, by using smaller imaging and recording 
elements such as a CCD, CID, or CMOS APS camera as disclosed in 
25 U.S. Serial No. 08/373,446, expressly incorporated herein by 

reference. Additionally, the distance between image planes I 702 
and I 703 of lenses 702 and 703, respectively, may be reduced to 
the point where the image planes coincide, further reducing the 
thickness of the camera body. 
30 Figure 3 discloses camera body 801, similar to that of 

camera body 701, and lenses 802 and 803 with image planes I 802 and 
I 803 , respectively, each having a f ield-of -view of exactly 180°. 
Lens 802 receives the image of hemisphere 804 and lens 803 

14 
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receives the image of hemisphere 805. Similar to Figure 2 above, 
the lenses attach to camera body 801 in a back-to-back 
arrangement where the lenses are mounted such that the image 
planes I 802 and I 803 from the lenses fall between each of the 
lenses and both lenses' optical axes A coincide in a single line 
which passes through each lens and camera. As discussed with 
reference to Figure 2 above, because camera body 801 has a 
thickness (i.e., the distance between lenses 802 and 803 is 
greater than zero) , the image capture system 800 has blind spots 
806 on the sides of the camera body 801. These blind spots may 
be reduced by decreasing the distance between lenses 802 and 803. 
Here, this means reducing the thickness of camera body 801. This 
may be accomplished, inter alia, by reducing the size of the 
imaging and recording components as discussed above in reference 
to Figure 2 . 

Referring now to Figure 4, two cameras 201 and 202 equipped 
with lenses 203, 204, each having a f ield-of -view (FOV) greater 
than 180° , are disclosed in a back-to-back arrangement (the image 
planes (not shown) falling between each of the lenses and the 
optical axes of the lenses 203 and 204 are collinear as 
designated by line A) . Because each camera 201, 202 has a lens 
(203, 204) which has a f ield-of -view (FOV) greater than 180°, 
each captures more than the image of a complete hemisphere. By 
employing two cameras in this arrangement, the camera system 
captures a complete spherical image. The types of cameras 
employed are chosen from the group comprising of at least still 
cameras with loaded film or digital image capture, motion picture 
cameras with loaded film or digital image capture, the KODAK™ 
digital image capture system, video, and linear scanning CID, 
CCD, or CMOS APS camera arrays. The outputs of cameras 201 and 
202 connect by means of electrical, optical, or electro-optical 
links 215 to hemispherical-to-spherical image converter 216. 
When the captured hemispherical images are stored on film, 

15 
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optical -to-electrical converter 215A converts the stored images 
into a form usable by hemispherical -to-spherical image converter 
216. Optical-to-electrical converter 215A includes a scanning 
system which scans a photographed image and outputs a high 
5 resolution, electronic replica of the photographed image. One 

converter includes the Kodak" Photo-CD Rom converter which takes 
a photograph and converts it into a high resolution digital form 
which then may be stored on a compact disk. Hemispherical-to- 
spherical converter 216 receives the hemispherical images from 
10 cameras 201 and 202 (or alternatively, from optical-to-electrical 
converter 215A) . 

«=• The cameras include additional features allowing for dual 

Ji image capture. For example, the backs of the cameras are 

!L -'f attached to each other via separable attachment devices 401. 

,jf.5 Attachment devices 401 may be locking pins, locking clasps, lever 

and clip systems, etc. Also, each camera's shutter release may 
; H] be controlled by a single button 402 A (common shutter release 

control) with either a mechanical or electrical servo linkage 
r2 releasing each camera's shutter. Additionally, to allow a 

t]JZ0 photographer to ensure his or her image is not recorded by the 
^ . spherical image capture system, the dual camera system includes a 
shutter auto timer or a remote shutter activation control 403 
controlling the common shutter release control, allowing the 
photographer to move to a concealed or non- image -captured 
25 position. The remote shutter control 403 may be an IR 

transmitter or remote shutter release cable. Further, the dual 
camera system may include two different shutters release control 
buttons 402B operable independently or sequentially. The 
sequential shutter operations allow the photographer to walk 
3 0 around to the other side of the dual camera system so as not to 
become part of the captured spherical image. 

Next, hemispherical-to-spherical converter 216 combines the 
hemispherical images into a single, complete spherical image. 
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Finally, the edges of the two hemispherical images may be 
combined to form a seamless spherical image. Removing the seams 
from the two hemispherical images may be accomplished in a number 
of ways. For example, the two images may be "airbrushed" 
together (where any difference between the two images at the 
periphery of the images are smoothed together. Alternatively, a 
more complex method of seaming the two images together may 
include matching related pixels by their luminance and 
chrominance values and interpolating the corresponding values for 
interstitial pixels. In the event that a partial overlap exists 
between the two hemispherical images, the converter processes the 
spherical image to remove the partial overlap any distortion and 
creates a single, complete, formed image. The processing may 
include choosing and displaying one hemisphere over the other, 
weighted and non-weighted averaging of the overlapping sections, 
and linear and non-linear approximations creating intermediary 
images . 

Figure 5A shows lenses 203 and 204 positioned remotely from 
cameras 201 and 202. Here, the image planes I 203 and I 204 fall 
between lenses 203 and 204 and the optical axes of the lenses 203 
and 204 are collinear as designated by line A. Electrical, 
optical (including fiber optic lines) , or electro-optical links 
215 connect the images received from lenses 2 03 and 204 to the 
cameras 201 and 202. Next, hemispherical -to- spherical image 
converter 216 receives the outputs from cameras 201 and 202 and 
outputs a spherical image as described in relation to Figure 4. 

Figure 5B shows single lens 2 03 positioned remotely from 
camera 201. Electrical, optical (including fiber optic lines), 
or electro-optical links 215 connect the image received from lens 
203 to the camera 201. Next, camera 201 captures a first 
hemispherical image. The output of camera 201 (a still or video 
image contained in a frame or frames of film, digital or analog 
signal) is sent to mirror image converter 901 and one input of 
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the hemispherical to spherical image converter 216. The mirror 
image converter 901 assumes many forms depending on the form of 
image relayed to it. For developed film, converter 901 refers to 
a re-scanning system re-scanning the developed film with the film 
flipped (flipped from a left to right orientation to a right to 
left orientation) . For an optical or electrical signal, 
converter 901 refers to a signal processing system which 
automatically creates a second hemispherical image from the first 
hemispherical image. The output of converter 901 flows to the 
hemispherical -to-spherical image converter 216 as the second 
hemispherical image. Finally, hemispherical-to-spherical image 
converter 216 outputs a spherical image as described in relation 
to Figure 4 . 

Figure 6A shows an alternative arrangement of the cameras 
201 and 202 and the lenses 203 and 204. The optical axes of the 
lenses 203 and 204 are collinear as designated by line A. Here, 
the devices used to convey the images from lenses 203 and 2 04 to 
cameras 201 and 2 02 include hollow chambers with reflective 
optics 215B and refractive optics 215C, as necessary for proper 
transmission of the hemispherical images. The reflective optics 
215B allow the cameras 201 and 202 to be moved from a location 
directly behind each lens 203, 204. The refractive optics 215C 
aid in focusing the hemispherical images generated by lenses 2 03 
and 2 04. This movement of the cameras from behind the lenses 
allows the lenses to be moved closer together, maximizing the 
area photographed. 

A further modification includes the substitution of the APS 
camera array of co-pending U.S. Application 08/373,446 (expressly 
incorporated herein by reference) for the optical system 
described above. Because of the small size of an APS camera 
array, two arrays may be placed back to back to further maximize 
the content of each hemispherical image. An advantage of using 
APS camera arrays is the shifted processing location of the 
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Omniview engine. Specifically, by adding additional processing 
circuitry on the APS camera array chip, the selection and 
"dewarping" transformations may be performed locally on the APS 
chip. This results in less subsequent processing of the image as 
5 well as a reduction in the bandwidth required for sending each 
hemispherical image to an external processing device. 

Furthermore, as described above, image conduits 215 may 
include optical fibers instead of the reflective optics 215B and 
refractive optics 215C. An imaging system including optical 
10 fibers connected between a hemispherical lens and imaging array 
is found in U.S. Patent No. 5,313,306 to Martin which is 
expressly incorporated by reference. The present invention 
~S includes the application of the spherical imaging system with a 
y combination of an endoscope and dual hemispherical lenses to 
ITS. capture hemispherical images of remote locations. Converter 216 
J3 combines the hemispherical images to a form complete, spherical 
;}f : image . 

si ly: 

Figure 6B relates to another embodiment where a single 
□ camera 201A captures the images produced by lenses 203 and 204. 
Up The optical axes of the lenses 203 and 204 are collinear as 
H designated by line A. Here, employing a single camera to 
capture both hemispherical images (from lenses 203 and 2 04) 
eliminates the bulk of the second camera. For example, where 
camera 201A is a still camera, the camera records the two 
25 hemispherical images in a single frame in a side-by-side 

relationship, exposed at the same time or during related time 
intervals. Alternatively, the two images may be captured in 
separate frames, exposed at the same time or during related time 
intervals. The same applies to video and motion picture cameras 
30 as well. Image capture with a single camera may be used in the 
other embodiments of described in greater detail herein. A 
system of Figure 6B including an APS camera array may be mounted 
onto a single, silicon chip. This combination has multiple 
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advantages including reduced size of the image capture system, 
reduced bulk from extra cameras, higher resolution from the APS 
camera arrays. 

Figure 7A shows first 205 and second 206 hemispherical 
images, each taken from one of cameras 201 or 202. Figure 7A 
also shows the edges 207, 208 (or seams) of each hemispherical 
image. Figure 7B shows the two images 205 and 206 combined into 
a single, spherical image 2 09. Seams 2 07 and 208 have been 
combined to form the single, seamless image 209. 

Figure 8 shows a possible future viewing system for viewing 
the formed spherical image system. The image planes I 203 and I 204 
fall between lenses 2 03 and 204 and the optical axes of the 
lenses 203 and 204 are collinear as designated by line A. Image 
input buffer 217 temporarily stores images received from cameras 
201 and 202 until hemispherical-to-spherical image converter 216 
accepts the stored images. Also, Figure 8 includes options for 
the spherical images. For example, after combining the two 
hemispherical images into a single, spherical image in converter 
216, the spherical image may be immediately viewed through 
viewing engine 218. Viewing engine 218 includes the Omniview 
calculation engine with viewer communication interface 124 as 
shown in Figure 1 of co-pending U.S. Serial No. 08/373,446 
(expressly incorporated herein by reference) . Here, the user may 
view selected portions of the formed spherical image as output 
from the hemispherical-to-spherical image converter 216. 
Alternatively, the spherical image may be stored in storage 
device 219. The storage device 119 may include video tape, CD- 
ROM, semiconductor devices, magnetic or magneto-optical disks, or 
laser disks as the storage medium. By the interconnections 
between viewing engine 218 and storage device 219, a new 
spherical image may be displayed and saved in storage device 219 
as well as saved in storage device 219 and viewed at a later 
time . 
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Further enhancements include using two side-by- side 
hemispherical lens equipped cameras for stereo-optical viewing. 
Additionally, the back-to-back camera system described herein may 
be attached to the exterior of any of a number of different 
5 vehicles for spherical image capture of a number of different 
environments . 



Captured Image Transformation 

Figures 9-13 relate to the captured image transformation 
10 system. 

In order to minimize the size of the camera orientation 
system while maintaining the ability to zoom, a camera 
O orientation system that utilizes electronic image transformation 

rather than mechanisms was developed. While numerous patents on 
Nt5 mechanical pan-and-tilt systems have been filed, no approach 

using strictly electronic transforms and 180° or greater field of 
yl view optics is known to have been successfully implemented. In 

iy addition, the electro-optical approach utilized in the present 

p invention allows multiple images to be extracted from the output 

Mlo of a signaled camera. These images can be then utilized to 
\* t energize appropriate alarms, for example, as a specific 

y? application of the basic image transformation in connection with 

^ a surveillance system. As utilized herein, the term 

"surveillance" has a wide range including, but not limited to, 
25 determining ingress or egress from a selected environment. 

Further, the term "wide angle" as used herein means a field-of- 
view of about eighty degrees or greater. Motivation for this 
device came from viewing system requirements in remote handling 
applications where the operating envelop of the equipment is a 
30 significant constraint to task accomplishment. 

The principles of the optical transform utilized in the 
present invention can be understood by reference to the system 10 
of Figures 9A and 9B. (This is also set forth in the aforecited 
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U.S. Patent Application Serial Number 07/699,366 that is 
incorporated herein by reference.) Referring to Figure 9A, shown 
schematically at 11 is a wide angle, e.g., a hemispherical, lens 
that provides an image of the environment with a 180 degree or 
5 greater f ield-of -view. The lens is attached to a camera 12 which 
converts the optical image into an electrical signal. These 
signals are then digitized electronically 13 and stored in an 
image buffer 14 within the present invention. An image 
processing system consisting of an X-MAP and a Y-MAP processor 
10 shown as 16 and 17, respectively, performs the two-dimensional 

transform mapping. The image transform processors are controlled 
by the microcomputer and control interface 15 . The microcomputer 
control interface provides initialization and transform parameter 
calculation for the system. The control interface also 
fJ5 determines the desired transformation coefficients based on 
.£ orientation angle, magnification, rotation, and light sensitivity 

input from an input means such as a joystick controller 22 or 
\~ computer input means 23. The transformed image is filtered by a 

C 2 -dimensional convolution filter 18 and the output of the 

jSo filtered image is stored in an output image buffer 19. The 
;U output image buffer 19 is scanned out by display electronics 20 

% to a video display device 21 for viewing. 

A range of lens types can be accommodated to support various 
fields of view. The lens optics 11 correspond directly with the 
25 mathematical coefficients used with the X-MAP and Y-MAP 

processors 16 and 17 to transform the image. The capability to 
pan and tilt the output image remains even though a different 
maximum f ield-of -view is provided with a different lens element. 
The invention can be realized by proper combination of a 
30 number of optical and electronic devices. The lens 11 is 

exemplified by any of a series of wide angle lenses from, for 
example, Nikon, particularly the 8mm F2 . 8 . Any video source 12 
and image capturing device 13 that converts the optical image 
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into electronic memory can serve as the input for the invention 
such as a Videk Digital Camera interfaced with Texas Instrument's 
TMS 34 061 integrated circuits. Input and output image buffers 14 
and 19 can be construed using Texas Instrument TMS44C251 video 
random access memory chips or their equivalents. The control 
interface can be accomplished with any of a number of 
microcontrollers including the Intel 80C196. The X-MAP and Y-MAP 
transform processors 16 and 17 and image filtering 19 can be 
accomplished with application specific integrated circuits or 
other means as will be known to persons skilled in the art. The 
display driver can also be accomplished with integrated circuits 
such as the Texas Instruments TMS34061. The output video signal 
can be of the NTSC RS-170, for example, compatible with most 
commercial television displays in the United States. Remote 
control 22 and computer control 23 are accomplished via readily 
available switches and/or computer systems than also will be well 
known. These components function as a system to select a portion 
of the input image (hemispherical or other wide angle) and then 
mathematically transform the image to provide the proper 
prospective for output. The keys to the success of the 
perspective correction system include: 

(1) the entire input image need not be transformed, only 
the portion of interest; 

(2) the required mathematical transform is predictable 
based on the lens characteristics; and 

(3) calibration coefficients can be modified by the end 
user to correct for any lens/camera combination supporting both 
new and retrofit applications. 

Figure 9B contains elements similar to that of Figure 9A but 
is implemented in a personal computer represented by dashed line 
D. The personal computer includes central processing unit 15 7 
performing the perspective correction algorithms X-MAP 16' and Y- 
MAP 17' as stored in RAM, ROM, or some other form. The display 
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driver 2 0 outputs the perspective corrected image to computer 
display monitor 21 ' . 

The transformation that occurs between the input memory 
buffer 14 and the output memory buffer 19, as controlled by the 
5 two coordinated buffer 19, as controlled by the two coordinated 
transformation circuits 16 and 17 of Figure 9A (or algorithms as 
stored in 16' and 17' of Figure 9B) , is better understood by 
referring to Figures 10 is a rendering of the image of a grid 
pattern produced by a hemispherical lens. This image has a 
10 f ield-of -view of 180 degrees and shows the contents of the 

environment throughout and entire hemisphere. Notice that the 
resulting image in Figure 10 is significantly distorted relative 
£ to human perception. Similar distortion will be obtained even 

w with lesser f ield-of -view lenses. Vertical grid lines in the 

JpL5 environment appear in the image plane as 24a, 24b, and 24c. 
^3 Horizontal grid lines in the environment appear in the image 

plane as 25a, 25b, and 25c, The image of an object is 
s exemplified by 26. A portion of the image in Figure 10 has been 

; t 5 corrected, magnified, and rotated to produce the image shown in 

f ; :|° Figure 11. Item 27 shows the corrected representation of the 
H object in the output display. The results shown in the image in 

: g Figure 11 can be produced from any portion of the image of Figure 

10 using the present invention. The corrected perspective of the 
view is demonstrated by the straightening of the grid pattern 
25 displayed in Figure 11. In the present invention, these 

transformations can be performed at real-time video rates (e.g., 
thirty times per second) , compatible with commercial video 
standards . 

The transformation portion of the invention as described has 
3 0 the capability to pan and tilt the output image through the 

entire f ield-of -view of the lens element by changing the input 
means, e.g. the joystick or computer, to the controller. This 
allows a large area to be scanned for information as can be 
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useful in security and surveillance applications. The image can 
also be rotated through any portion of 360 degrees on its axis 
changing the perceived vertical of the displayed image. This 
capability provides the ability to align the vertical image with 
5 the gravity vector to maintain a proper perspective in the image 
display regardless of the pan or tilt angle of the image. The 
invention also supports modifications in the magnification, used 
to display the output image. This is commensurate with a zoom 
function that allows a change in the f ield-of -view of the output 
10 image. This function is extremely useful for inspection and 
surveillance operations. The magnitude of zoom provided is a 
function of the resolution of the input camera, the resolution of 
3 the output display, the clarity of the output display, and the 

W- amount of picture element (pixel) averaging that is used in a 

jjf5 given display. The invention supports all of these functions to 
uJ provide capabilities associated with traditional mechanical pan 

(through 180 degrees) , tilt (through 180 degrees) , rotation 
= (through 360 degrees), and zoom devices. The digital system also 

^ supports image intensity scaling that emulates the functionality 

^0 of a mechanical iris by shifting the intensity of the displayed 
H image based on commands from the user or an external computer. 

The postulates and equations that follow are based on the 
image transformation portion of the present invention utilizing a 
wide angle lens as the optical element. These also apply to 
25 other f ield-of -view lens systems. There are two basic properties 
and two basic postulates that describe the perfect wide angle 
lens system. The first property of such a lens is that the lens 
has a 27T ateradian f iled-of -view and the image it produces is a 
circle. The second property is that all objects in the f ield-of - 
3 0 view are in focus, i.e. the perfect wide angle lens has an 

infinite depth-of -field. The two important postulates of this 
lens system (refer to Figures 12 and 13) are stated as follows: 
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Postulate 1: Azimuth angle invariability - For object 
points that lie in a content plane that is perpendicular to the 
image plane and passes through the image plane origin, all such 
points are mapped as image points onto the line of intersection 
5 between the image plane and the content plane, i.e. along a 

radial line. The azimuth angle of the image points is therefore 
invariant to elevation and object distance changes within the 
content plane . 

Postulate 2: Equidistant Projection Rule - The radial 

10 distance, r, from the image plane origin along the azimuth angle 
containing the projection of the object point is linearly 
proportional to the zenith angle /?, where (3 is defined as the 

4) angle between a perpendicular line through the image plane origin 

^ and the line from the image plane origin to the object point. 

:;15 Thus the relationship: 

^ r = kjS (1) 

Using these properties and postulates as the foundation of 
i;f0 the lens system, the mathematical transformation for obtaining a 
r ^ perspective corrected image can be determined. Figure 12 shows 
the coordinate reference frames for the object plane and the 
image plane. The coordinates u,v describe object points within 
the object plane. The coordinates x,y,z describe points within 
25 the image coordinate frame of reference* 

The object plane shown in Figure 12 is a typical region of 
interest to determine the mapping relationship onto the image 
plane to properly correct the object. The direction of view 
vector, DOV[x,y,z], determines the zenith and azimuth angles for 
3 0 mapping the object plane, UV, onto the image plane, XY. The 
object plane is defined to be perpendicular to the vector, 
DOV[x,y,z] . 
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The location of the origin of the object plane in terms of 
the image plane [x,y,z] in spherical coordinates is given by: 



y = D sin/3 cosd 
z = D cos/3 

where D = scaler length from the image plane origin to the object 
plane origin, /3 is the zenith angle, and d is the azimuth angle 
in image plane spherical coordinates. The origin of object plane 
is represented as a vector using the components given in Equation 
1 as : 

DOV[x,y,z] = [ Dsin/3cos3, Dsin^sinS, Dcos/S] (3) 

DOV[x,y,z] is perpendicular to the object plane and its 
scaler magnitude D provides the distance to the object plane. By 
aligning the XY plane with the direction of action of DOV[x,y,z], 
the azimuth angle d becomes either 90 or 270 degrees and 
therefore the x component becomes zero resulting in the 
DOV [x, y , z] coordinates : 

DOV[x,y,z] = [ 0, -Dsin/3, DcosjS] (4) 

Referring now to Figure 13, the object point relative to the 
UV plane origin in coordinates relative to the origin of the 
image plane is given by the following: 



z = v sin/? 

therefore, the coordinates of a point P(u,v) that lies in the 
object plane can be represented as a vector P[x,y,z] in image 
plane coordinates : 



x = D sin/? cosd 



(2) 



x = u 



y = v cos/3 



(5) 



P[x,y,z] = [ u, vcos/3, vsin/3] 



(6) 
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where P[x,y,z] describes the position of the object point in 
image coordinates relative to the origin of the UV plane. The 
object vector o[x,y,z] that describes the object point in image 
coordinates is then given by: 

Ofx,y,z] = DGV[x,y,z] + P[x,y,z] (7) 
0[x,y,z] = [u, voosp - DsinP, vsinp + Dcosp] (3) 



Projection onto a hemisphere of radius R attached to the image 
plane is determined by scaling the object vector o[x,y,z] to 
produce a surface vector s[x,y,z]: 



* lwJ - 10[x,y,z]i 



!9) 



By substituting for the components of o[x,y,z] from Equation 
8, the vector S[x,y,z] describing the image point mapping onto 
the hemisphere becomes: 

ROf u, (vcosfl-DsinB), (vsinfl+Dcosfl)] 
S[x t y,z]« _ - — [1Q) 

y u 2 +(vcosB-DsinBr+(vsinB+Dcosfir 

The denominator in Equation 10 represents the length or 
absolute value of the vector o[x,y,z] and can be simplified 
through algebraic and trigonometric manipulation to give: 

S[Xtyj2] . ROfu, (vcosB-DsinS), (vsinfi+Dcosfi)] 

V u 2 +v 2 +D 2 

From Equation 11, the mapping onto the two-dimensional image 
plane can be obtained for both x and y as: 

Ru 

X= / 9 7 f ~ (12) 

y u 2 +v 2 +d 2 
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y R(vcosfi-Dsinitt (i3) 
V u 2 +v 2 +D 2 

Additionally, the image plane center to object plane 
distance D can be represented in terms of the image circular 
5 radius R by the relation: 

D=mR (14) 

where m represents the scale factor in radial units R from 
10 the image plane origin to the object plane origin. Substituting 
Equation 14 into Equations 12 and 13 provides a means for 
obtaining an effective scaling operation or magnification which 
^ can be used to provide zoom operation. 

3 Ru 

y- / 9 (is) 

f- y u 2 +v 2 +m 2 R 2 

$ Using the equations for two-dimensional rotation of axes for 

M20 both the UV object plane and the XY image plane the last two 
;i| equations can be further manipulated to provide a more general 

Hj set of equations that provides for rotation within the image 

plane and rotation within the object plane. 

R[ uA-vB+mRsinfisind ] 
y u 2 +v 2 +m 2 R 2 

_ R[ uC-vD-mRsinBcos9 1 (18) 
V u 2 +v 2 +m 2 R 2 

where : 

30 A = (cos0cosd - sin0sindcos/3) 

B = (sin0cos<3 4- cos0sindcosj8) (19) 
C = (cos0sin3 + sin0cosdcos|6) 
D = (sin0sin3 - cos0cosdcosj3) 
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and where : 

R = radius of the image circle 
|8 = zenith angle 

d = Azimuth angle in image plane 
0 = Object plane rotation angle 
m = Magnification 

u,v = object plane coordinates 

x,y = image plane coordinates 

The Equations 17 and 18 provide a direct mapping from the UV 
space to the XY image space and are the fundamental mathematical 
result that supports the functioning of the present 
omnidirectional viewing system with no moving parts. By knowing 
the desired zenith, azimuth, and object plane rotation angles and 
the magnification, the locations of x and y in the imaging array 
can be determined. This approach provides a means to transform 
an image from the input video buffer to the output video buffer 
exactly. Also, the image system is completely symmetrical about 
the zenith, therefore, the vector assignments and resulting signs 
of various components can be chosen differently depending on the 
desired orientation of the object plane with respect to the image 
plane. In addition, these postulates and mathematical equations 
can be modified for various lens elements as necessary for the 
desired f ield-of -view coverage in a given application. 

The input means defines the zenith angle, (3, the azimuth 
angle, d, the object rotation, 0, and the magnification, m. 
These values are substituted into Equations 19 to determine 
values for substitution into Equations 17 and 18. The image 
circle radius, R, is fixed value that is determined by the camera 
lens and element relationship. The variables u and v vary 
throughout the object plane determining the values for x and y in 
the image plane coordinates . 
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From the foregoing, it can be seen that a wide angle lens 
provides a substantially hemispherical view that is captured by a 
camera. The image is then transformed into a corrected image at 
a desired pan, tilt, magnification, rotation, and focus based on 
5 the desired view as described by a control input. The image is 
then output to a television display with the perspective 
corrected. Accordingly, no mechanical devices are required to 
attain this extensive analysis and presentation of the view of an 
environment through 180 degrees of pan, 180 degrees of tilt, 360 
10 degrees of rotation, and various degrees of zoom magnification. 

As indicated above, one application for the perspective 
correction of images obtained with a motionless wide angle camera 
O is in the field of surveillance. The term "surveillance" is 

•rl meant to include inspection and like operations as well. It is 

LIB often desired to continuously or periodically view a selected 
*H environment to determine activity in that environment. The term 

fri "environment" is meant to include such areas as rooms, 

warehouses, parks and the like. This activity might be, for 
H example, ingress and egress of some object relative to that 

I ; i0 environment. It might also be some action that is taking place 

in that environment. It may be desired to carry out this 
ig surveillance either automatically at the desired frequency (or 

; ^ continuously), or upon demand by an operator. The size of the 

environment may require more than one motionless camera for 
25 complete surveillance. 

While a preferred embodiment has been shown and described, 
it will be understood that it is not intended to limit the 
disclosure, but rather it is intended to cover all modifications 
and alternate methods falling within the spirit and the scope of 
3 0 the invention as defined in the appended claims. All of the 

above referenced U.S. patents and pending applications referenced 
herein are expressly incorporated by reference. 

Having thus described the aforementioned invention, 



31 



Application of McCall et ah 

We claim; 

1. A method for capturing a spherical image comprising the 
steps of: 

capturing a first hemispherical image via a first 
camera including a first hemispherical lens; 

capturing a second hemispherical image via a second 
camera including a second oppositely directed hemispherical lens; 
and, 

combining said first and second oppositely directed 
hemispherical images to create a spherical image. 

2 . The method according to claim 1 further including the 
step of transforming the spherical image to a perspective 
corrected spherical image. 

3. The method according to claim 2, said transforming step 
occurring within a computer and said method further including the 
step of displaying the perspective corrected spherical image on a 
display of said computer. 

4. A method for capturing and generating a spherical image 
comprising the steps of: 

capturing a first image via a camera including a lens 
having a f ield-of -view of at least 180°; 

creating a second image corresponding to a mirror image 
of the first image; and, 

combining said first and second images to create a 
spherical image . 

5 . The method according to claim 4 further including the 
step of transforming the spherical image to a perspective 
corrected spherical image. 

6. The method according to claim 5, said transforming step 
occurring within a computer and said method including the step of 
displaying the perspective corrected spherical image on a display 
of said computer. 
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7. An apparatus for capturing a spherical image 
comprising: 

a first camera equipped with a first 180° or greater 
f ield-of -view lens, said first camera and said first lens 
directed in a first direction, said first camera capturing a 
first image; 

a second camera with a second 180° or greater field-of- 
view lens, said second camera and said second lens directed in a 
second direction opposite said first direction, said first and 
second cameras arranged in a back-to-back position, said second 
camera capturing a second image; and, 

a combining system for combining said first and second 
images into a formed image. 

8. The apparatus according to claim 7, said first and 
second cameras are single-use cameras whose backs are attached to 
each other through separable locking devices. 

9. The apparatus according to claim 7, said first image is 
a 180° or greater f ield-of -view image and said second image is a 
180° or greater f ield-of -view image. 

10. The apparatus according , to claim 9, said formed image 
is a complete spherical image. 

11. The apparatus of claim 7, said first and second 
hemispherical lenses are optically connected to said first and 
second cameras through reflective optics. 

12. The apparatus according to claim 7, wherein said first 
and second cameras are still cameras. 

13. The apparatus according to claim 7, wherein said first 
and second cameras are video cameras . 

14. The apparatus according to claim 7, wherein said first 
and second cameras are motion picture cameras. 

15. The apparatus according to claim 7, wherein said first 
and second cameras are linear scanning CID camera arrays. 
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16. The apparatus according to claim 7, wherein said first 
and second cameras are linear scanning CCD camera arrays. 

17. The apparatus according to claim 7, wherein said first 
and second cameras are linear scanning CMOS APS camera arrays. 

18. The apparatus according to claim 7, said first and 
second cameras including film exposed to said first and second 
images, said apparatus further comprising an optical -to- 
electrical converter converting said first and second images 
stored on said film into a form usable by said combining system. 

19. The apparatus according to claim 7, said first and 
second hemispherical lenses connected to said first and second 
cameras though fiber optic lines. 

20. The apparatus according to claim 7, said combining 
system for combining said first and second images further 
processes any overlap between said first and second hemispherical 
images to remove any edge distortion in creating said formed, 
spherical image . 

21. An apparatus for capturing and forming a spherical 
image comprising: 

a camera equipped with a 180° or greater f ield-of -view 
lens, said first camera capturing a first image; 

a mirror image converter for converting said first 
image into a second image corresponding to a mirror image of the 
first image; and, 

a combining system for combining said first and second 
images into a spherical image . 

22. The apparatus of claim 21 where the camera is a single- 
use camera. 

23. The apparatus of claim 21 including a transformer which 
transforms a selected portion of said spherical image to a 
perspective corrected spherical image. 

24. The apparatus according to claim 23, said transformer 
disposed within a computer and containing hardwired perspective 
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correction algorithms and said computer including a display for 
displaying said perspective corrected spherical image. 

25. The apparatus according to claim 23, said transformer 
disposed within a computer, which contains perspective correction 
algorithms in software, and said computer including a display for 
displaying said perspective corrected spherical image. 

26. A system for capturing a spherical image comprising: 
a first hemispherical lens including a first image 

plane and a first optical axis, said first hemispherical lens 
directed in a first direction; 

a second hemispherical lens, said second lens including 
a second optical axis and a second image plane, said second lens 
directed in a second direction opposite said first direction such 
that the first and second image planes fall between the first and 
second lenses and the first and second optical axes are 
collinear; 

a first image capture device capturing an image from 
said first hemispherical lens; 

a second image capture device capturing an image from 
said second hemispherical lens; 

a combining system for combining said first and second 
images into a formed image. 

27. The system according to claim 26, said first image is a 
hemispherical image and said second image is a hemispherical 
image . 

28. The system according to claim 27, said formed image is 
a complete spherical image. 

29. The apparatus according to claim 26, wherein said first 
and second cameras are still cameras. 

30. The apparatus according to claim 26, wherein said first 
and second cameras are video cameras . 

31. The apparatus according to claim 26, wherein said first 
and second cameras are motion picture cameras. 
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32. The^ apparatus according to claim 26, wherein said first 

> r -. 

and second cameras are linear scanning CID camera arrays. 

33. The apparatus according to claim 26, wherein said first 
and second cameras are linear scanning CCD camera arrays. 

34. The apparatus according to claim 26, wherein said first 
and second cameras 'are linear scanning CMOS APS camera arrays. 

35. An apparatus for capturing a spherical image 
comprising: 

a first hemispherical lens having a first image plane 
and a first optical axis, said first lens directed in a first 
direction; 

a second hemispherical lens having a second image plane 
and a second optical axis, said second lens directed in a second 
direction opposite said first direction such that the first and 
second image planes fall between the f irst^ and second lenses and 
the first and second optical axes are collinear; 

a camera capturing a first image from said first 
hemispherical lens and capturing a second image from said second 
hemispherical lens; and, 

a combining system for combining said first and second 
images into a formed image. 
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Abstract 

A method and apparatus for capture of a spherical image is 
disclosed. The present invention includes at least one camera 
having a lens with at least a 180° f ield-of -view for capturing a 
hemispherical image. In a first embodiment, a second 
hemispherical image is created corresponding to a mirror image of 
the hemispherical image captured by the camera. In a second 
embodiment, two back-to-back cameras capture first and second 
hemispherical images, respectively. In both embodiments, a 
converter combines the two images along their outside edges to 
form a single, spherical image. Finally, the converter stores 
the complete spherical image for later retrieval and perspective 
corrected viewing. 
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As the below named inventor, we hereby declare that: 



Our residence, post office address and citizenship are as stated below next to our names; 

We believe we are the original, first and joint inventors of the subject matter which is claimed and for which a patent 
is sought on the invention entitled Method and Apparatus for Simultaneous Capture of a Spherical Image , 

the specification of which 

M is attached hereto. 

□ was filed on as Application Serial Number and was amended on 

(if applicable!. 

We hereby state that we have reviewed and understand the contents of the above identified specification, including 
the claims, as amended by any amendment referred to above. 

We acknowledge the duty to disclose information which is material to the examination of this application in accordance 
with Title 37, Code of Federal Regulations, § 1.56(a). 

Prior Foreign Application(s) 

We hereby claim foreign priority benefits under Title 35, United States Code, §1 19 of any foreign appiication(s) for 
pajtent or inventor's certificate listed below and have also identified below any foreign application(s) for patent or inventor's 
certificate having a filing date before that of the application on which priority is claimed: 



■ y 'ik\: ; Country 


Application Number 


Date of Filing . 
(day, montJ?< yearj 


Date of Issue 
{day* month r year) 


Priority Claimed 
Under 35 U-S.C. §11$ 

































W Prior United States Application (s) 

5 *f We hereby claim the benefit under Title 35, United States Code, § 1 20 of any United States applications) listed below and, 
insofar as the subject matter of each of the claims of this application is not disclosed in the prior United States application in 
theSjhanner provided by the first paragraph of Title 35, United States Code, § 1 1 2, we acknowledge the duty to disclose material 
information as defined in Title 37, Code of Federal Regulations, § 1.56(a) which occurred between the filing date of the prior 
application and the national or PCT international filing date of this application: 



Application Serial 
Number 


Date of Filing 

{OW, Month, Year} 


Status — Patented, 
Pending t Abandoned 


08/386,912 


February 8, 1995 


Pending 


08/339,633 


November 1 1, 1994 


Pending 


08/189,585 


January 31, 1994 


U.S. Patent No. 5,384,558 


08/014,508 


February 8, 1993 


US. Patent No. 5,359,363 


07/699,366 


May 13, 1991 


U.S. Patent No. 5,185,667 


08/373,446 


January 17, 1995 


Pending 



And we hereby appoint, both jointly and severally, as our attorneys with full power of substitution and revocation, to 
prosecute this application and to transact all business in the Patent and Trademark Office connected herewith the following 
attorneys who are all members of the Bar of the District of Columbia, their registration numbers being listed after their names: 



Donald W. Banner, Registration No. 17,037; Harold J. Birch, Registration No. 16,527; Edward F. McKie, Jr., Registration 
No. 17,335; William W. Beckett, Registration No. 18,262; Dale H. Hoscheit, Registration No. 19,090; Joseph M. Potenza, 
Registration No. 28,175; Alan I, Cantor, Registration No. 28,163; James A. Niegowski, Registration No. 28,331; Barry L. 
Grossman, Registration No. 30,844; Joseph M. Skerpon, Registration No. 29,864; Thomas L. Peterson, Registration No. 
30,969; Nina L. Mediock, Registration No. 29,673; William J. Fisher, Registration No. 32,133; and Thomas H. Jackson, 
Registration No. 29,808. 



A!;. cc rr espcridcnc« .::;J telephone communications should be artciress*^ to Banner, Birch, McKie anc* Beckett. Ftevp^h 
Floor, 1001 G Street, N.W., Washington, D.C. 20001-4597, telephone number (202) 508-9100, which is also the address and 
telephone number of each of the above listed attorneys. 

We hereby declare that all statements made herein of our own knowledge are true and that all statements made on 
information and belief are believed to be true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such wijiful false statements mayjeopajrdj^e^hy validity of the application or any patent issuing thereon. 
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We believe we are the original, first and joint inventors of the subject matter which is claimed and for which a patent 
is sought on the invention entitled Method and Apparatus for Simultaneous Capture of a Spherical Image , 

the specification of which 
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□ was filed on as Application Serial Number and was amended on 
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